Faster SVD-Truncated Least-Squares Regression

We develop a fast algorithm for computing the “SVD-truncated” regularized solution to the leastsquares problem: minx ‖Ax − b‖2. Let Ak of rank k be the best rank k matrix computed via the SVD of A. Then, the SVD-truncated regularized solution is: xk = A † k b. If A is m × n, then, it takes O(mnmin{m,n}) time to compute xk using the SVD of A. We give an approximation algorithm for xk which constructs a rank-k approximation Ãk and computes x̃k = Ã † kb in roughly O(nnz(A)k logn) time. Our algorithm uses a randomized variant of the subspace iteration. We show that, with high probability: ‖Ax̃k − b‖2 ≈ ‖Axk − b‖2 and ‖xk − x̃k‖2 ≈ 0.